Pose synthesis in unseen human poses

ABSTRACT

Techniques related to synthesizing an image of a person in an unseen pose are discussed. Such techniques include detecting a body part occlusion for a body part in a representation of the person in a first image and, in response to the detected occlusion, projecting a representation of the body part from a second image having a different view into the first image. A geometric transformation based on a source pose of the person and a target pose is then applied to the merged image to generate a synthesized image comprising a representation of the person in the target pose.

BACKGROUND

In some contexts, translating a new pose to a person in an input imagehaving a different pose is advantageous. Such new pose images may beused for entertainment, for presentation to an audience, or for avariety of other uses. Currently, machine learning may be used togenerate new pose images based on a person in an input image and thedesired new pose. Such techniques, however, address the translation ofonly a single person at a time in contexts that have clean backgrounds.

It is desirable to translate new poses to humans in input images thatinclude more than one person, potentially occluding one another, and inreal life contexts that include complicated background information. Suchnew pose images may be used in a wide variety of contexts. It is withrespect to these and other considerations that the present improvementshave been needed. Such improvements may become critical as the desire toprovide new and immersive user experiences in imaging and video becomesmore widespread.

BRIEF DESCRIPTION OF THE DRAWINGS

The material described herein is illustrated by way of example and notby way of limitation in the accompanying figures. For simplicity andclarity of illustration, elements illustrated in the figures are notnecessarily drawn to scale. For example, the dimensions of some elementsmay be exaggerated relative to other elements for clarity. Further,where considered appropriate, reference labels have been repeated amongthe figures to indicate corresponding or analogous elements. In thefigures:

FIG. 1 illustrates an example apparatus for generating an image of aperson in a synthetic pose;

FIG. 2 illustrates an example pose estimation module for implementationin the apparatus of FIG. 1;

FIG. 3 illustrates an example camera array trained on an example scene;

FIG. 4 illustrates an example input image from a particular view of anexample scene;

FIG. 5 illustrates an example background/foreground mask;

FIG. 6 illustrates example pose data corresponding to a number ofpersons in an example input image region;

FIG. 7 illustrates an example body part image corresponding to anexample input image portion;

FIG. 8 illustrates an example body part region having an externalocclusion through a portion thereof;

FIG. 9 illustrates an example process for detecting occlusions in a bodypart image;

FIG. 10 illustrates an example geometric transformation to determine asynthesized pose image;

FIG. 11 is a flow diagram illustrating an example process for generatingan image of a person in a synthesized pose;

FIG. 12 is an illustrative diagram of an example system for generatingan image of a person in a synthesized pose;

FIG. 13 is an illustrative diagram of an example system; and

FIG. 14 illustrates an example device, all arranged in accordance withat least some implementations of the present disclosure.

DETAILED DESCRIPTION

One or more embodiments or implementations are now described withreference to the enclosed figures. While specific configurations andarrangements are discussed, it should be understood that this is donefor illustrative purposes only. Persons skilled in the relevant art willrecognize that other configurations and arrangements may be employedwithout departing from the spirit and scope of the description. It willbe apparent to those skilled in the relevant art that techniques and/orarrangements described herein may also be employed in a variety of othersystems and applications other than what is described herein.

While the following description sets forth various implementations thatmay be manifested in architectures such as system-on-a-chip (SoC)architectures for example, implementation of the techniques and/orarrangements described herein are not restricted to particulararchitectures and/or computing systems and may be implemented by anyarchitecture and/or computing system for similar purposes. For instance,various architectures employing, for example, multiple integratedcircuit (IC) chips and/or packages, and/or various computing devicesand/or consumer electronic (CE) devices such as set top boxes, smartphones, etc., may implement the techniques and/or arrangements describedherein. Further, while the following description may set forth numerousspecific details such as logic implementations, types andinterrelationships of system components, logic partitioning/integrationchoices, etc., claimed subject matter may be practiced without suchspecific details. In other instances, some material such as, forexample, control structures and full software instruction sequences, maynot be shown in detail in order not to obscure the material disclosedherein.

The material disclosed herein may be implemented in hardware, firmware,software, or any combination thereof. The material disclosed herein mayalso be implemented as instructions stored on a machine-readable medium,which may be read and executed by one or more processors. Amachine-readable medium may include any medium and/or mechanism forstoring or transmitting information in a form readable by a machine(e.g., a computing device). For example, a machine-readable medium mayinclude read only memory (ROM); random access memory (RAM); magneticdisk storage media; optical storage media; flash memory devices;electrical, optical, acoustical or other forms of propagated signals(e.g., carrier waves, infrared signals, digital signals, etc.), andothers.

References in the specification to “one implementation”, “animplementation”, “an example implementation”, etc., indicate that theimplementation described may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Moreover, such phrasesare not necessarily referring to the same implementation. Further, whena particular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the art to effect such feature, structure,or characteristic in connection with other implementations whether ornot explicitly described herein.

The terms “substantially,” “close,” “approximately,” “near,” and“about,” generally refer to being within +/−10% of a target value. Forexample, unless otherwise specified in the explicit context of theiruse, the terms “substantially equal,” “about equal” and “approximatelyequal” mean that there is no more than incidental variation betweenamong things so described. In the art, such variation is typically nomore than +/−10% of a predetermined target value. Unless otherwisespecified the use of the ordinal adjectives “first,” “second,” and“third,” etc., to describe a common object, merely indicate thatdifferent instances of like objects are being referred to, and are notintended to imply that the objects so described must be in a givensequence, either temporally, spatially, in ranking or in any othermanner.

Methods, devices, apparatuses, computing platforms, and articles aredescribed herein related to generating an image of a person in asynthetic pose and, in particular, to rectifying body part occlusions inthe generation of an image of a person in a synthesized pose.

As described above, it may be advantageous to translate a target pose toa person that has not been imaged in the target pose. For example, animage of a person in a source pose that is not the target pose may beused to generate a synthesized image of the person in the target posesuch that an image of a person in an unseen pose is generated. As usedherein, the term unseen or unseen pose indicates the image has not beenobtained by an image capture device (e.g., camera) but was insteadgenerated or synthesized. Similarly, the term synthesized pose orsynthesized image indicates a pose or image that was not obtained by animage capture device but was instead generated. Such synthesized posesor synthesized images may be used in a variety of contexts. For example,the synthesized poses or synthesized images may be presented to viewersfor entertainment purposes. In some embodiments, the synthesized posesor synthesized images may be used as a coaching tool to provideadvantages to coaches in sporting context. In some embodiments, thesynthesized poses or synthesized images may be presented in a videogaming context. Furthermore, the synthesized poses or synthesized imagesmay be used to generate point clouds inclusive of the unseen poses togenerate immersive content (e.g., 360 views from any location) within amodeled 3D environment for the purpose of entertainment.

The techniques discussed herein provide synthesized poses in complexscenarios with multiple people (e.g., multiple players in a sportingevent), dynamic backgrounds, and external occlusions. Furthermore, thedisclosed techniques may advantageously be applied in real worldcontexts (e.g., based on images attained from a sporting orentertainment event) using images that have relatively low resolutionwith many details of the attained poses and the background being blurredout as the image capture system is necessarily at a large distance fromthe scene being captured.

As discussed further herein below, in some embodiments, based on aninput image corresponding to a view of a scene, a segmentation operationis performed to separate background and foreground, and to isolate oneor more persons in the image. Such processing may be performed on anynumber of images corresponding to different views of the scene. In someembodiments, to synthesize a background, neighboring pixels are used toartificially generate a background. Furthermore, given the discussedperson isolation and a target pose, a generative adversarial network(GAN) may be used for a geometric transformation of the personrepresented in the original image to synthesize an image of the personin the target pose. In some embodiments, the GAN is trained usingtraining data specific to the event for which image capture is beingperformed. For example, the GAN may be trained for a particular sport(e.g., soccer, football, basketball, etc.), for a particular match up ina particular sport, for a particular entertainment event, etc. In someembodiments, the GAN or preprocessing for the GAN includes imagere-sizing to resolve the issue of poor resolution of the persons. Asdiscussed, for example, although high resolution cameras are used toattain images of a scene, due to the nature of the event being captured,the cameras are necessarily far from the individual people being imagedand provide low resolution images. Furthermore, in case of externalocclusions and self-occlusions of body parts, images from alternativelocations are used. In some embodiments, in an occluded scenario (e.g.,which would lead to poor pose estimation), an image from a differentcamera view re-projected back to the original or target view using thecamera calibration matrices.

The reprojected body part may then be used for the geometrictransformation discussed above. The resultant synthesized image pose maybe used in any context as discussed herein. The synthesized image posegenerated using the techniques discussed herein offer high image qualityand high pose quality in complex implementations. Notably, theprocessing pipeline discussed herein may use highly accurate labeledinformation in a sporting or entertainment context by labeling eachplayer pose estimation and training a network to understand playerposes. Using such techniques, the disclosed techniques may synthesizeposes to generate unseen scenarios that enrich the entertainmentexperience and/or provide understanding of player behavior that canpotentially benefit coaching and player simulation in the sporting andgaming industries.

FIG. 1 illustrates an example apparatus 100 for generating an image of aperson in a synthetic pose, arranged in accordance with at least someimplementations of the present disclosure. As shown in FIG. 1, apparatus100 may include a camera array 101, a segmentation module 102, a poseestimation module 103, and a pose synthesizer 104. Apparatus 100 may beimplemented in any suitable form factor device or one or more of suchdevices including a server computer, a cloud computing environment,personal computer, a laptop computer, a tablet, a phablet, a smartphone, a gaming console, a wearable device, a display device, anall-in-one device, a two-in-one device, or the like. Notably, in someembodiments, camera array 101 may be implemented separately from adevice implementing the remaining components of apparatus 100. Theimages captured via camera 101, input images 111, include simultaneouslycaptured images of a scene 110. As used herein, the term simultaneouslycaptured images indicates images that are synchronized to be captured atthe same or nearly the same time instance within a tolerance such as 0.1second. In some embodiments, the captured images are captured assynchronized captured video. For example, the components of apparatus100 may be incorporated into any multi-camera multi-processor system todeliver immersive visual experiences for viewers of a scene.

FIG. 2 illustrates an example pose estimation module 103 forimplementation in apparatus 100, arranged in accordance with at leastsome implementations of the present disclosure. As shown in FIG. 2, poseestimation module 103 may include a body part isolation module 201, anocclusion detection module 202, an alternative view selector 203, are-projection and merge module 204, a geometric transform module 205, aninterpolation and background merge module 206, and an input images andbody part images memory store 207. Pose estimation module 103 receives,for example, pose data 113, a target pose 114, and one or more of inputimages 111, and pose estimation module 103 generates a synthesized poseimage 115, as is discussed further herein below.

Returning to FIG. 1, apparatus 100 generates or otherwise attains inputimages 111. Input images 111 may include any suitable image data,picture data, video frame data, or the like or any data structurerepresentative of a picture at any suitable resolution. In anembodiment, each image of input images 111 includes RGB image data eachhaving R (red), G (green), and B (blue), values for pixels thereof;however any suitable image data format may be used such as YUV, YCbCr,etc. In an embodiment, input images 111 are pictures of sequences ofvideo pictures captured from different viewpoints. In an embodiment,input images 111 have 5 k resolution (e.g., a horizontal resolution ofaround 5,000 pixels such as 5120×2880 pixel resolution). In someembodiments, input images 111 have a resolution of not less than 4 k(e.g., a horizontal resolution of around 4,000 pixels and not less than3840 pixels such as 3840×2160 pixel resolution or 4096×2160 pixelresolution). As discussed, input images 111 include any number ofsimultaneously captured images of scene 110 such that images of scene110 are captured at the same or approximately the same time instance andsuch image capture is repeated at a particular frame rate (e.g., 60 fpsor 30 fps) over time to provide simultaneously attained video of scene110. Scene 110 may include any scene such as a sporting event, anentertainment event, a political event, etc. Although typically providedfor a high profile event, apparatus 100 may be applied to any scene 110.

As shown, segmentation, pose estimation, and pose synthesis may beapplied to any image or images of input images 111. Furthermore, thepose estimation and pose synthesis processing may be applied to anyperson or persons within an image. Notably, such processing may beapplied to a single person in an image or to any number of persons in asingle image or any number of other images. As used herein, the termrepresentation or representative of in the context of an image indicatesa representation of a particular object, person, entity, etc. isincluded in the image.

FIG. 3 illustrates an example camera array 101 trained on an examplescene 110, arranged in accordance with at least some implementations ofthe present disclosure. In the illustrated embodiment, camera array 101includes 36 cameras trained on a sporting field. However, camera array101 may include any suitable number of cameras trained on scene 110 suchas not less than 20 cameras. For example, camera array 101 may betrained on scene 110 to generate a 3D model of scene 110 and fewercameras may not provide adequate information to generate the 3D model.Camera array 101 may be mounted to a stadium (not shown) surrounding thesporting field of scene 110, calibrated, and trained on scene 110 tocapture simultaneous images or video. As shown, each camera of cameraarray 101 has a particular view of scene 110. For example, camera 301has a first view of scene 110 and camera 302 has a second view of ascene and so on. As used herein, the term view indicates the imagecontent of an image plane of a particular camera of camera array 101 orimage content of any view from a virtual camera located within scene110. Notably, the view may be a captured view (e.g., a view attainedusing image capture at a camera) such that multiple views includerepresentations of the same person, object, entity, etc.

Returning to FIG. 1, segmentation module 102 receives an image of inputimages 111 and generates a background/foreground (BG/FG) mask 112corresponding to the image of input images 111. Such segmentation may beperformed using any suitable technique or techniques. In an embodiment,a convolutional neural network (CNN) is applied to the image of inputimages 111 to generate BG/FG mask 112. For example, pixel values forthree color channels of the image may be stacked and fed to a pretrainedCNN to generate an output having a single channel indicating, for eachpixel, whether the pixel is part of the foreground (e.g., a value of 1)or the pixel is part of the background (e.g., a value of 0). In someembodiments, up-sampling of the input image is performed prior toapplication of the CNN. For example, in the case of resolutionconstraints, up-sampling is applied as necessary.

BG/FG mask 112 detects, for example, players and other pertinent objects(e.g., a ball) from a background, which may include a sporting field.Although discussed and illustrated herein with respect to sportingcontexts, input images 111 may include images of any suitable scene. Insome embodiments, the pretrained CNN is pretrained based on pretrainedimage data (marked with ground truth BG/FG information) pertinent toinput images 111 at any level of granularity. For example, thepretrained CNN may be pretrained using training images from the samesporting event being imaged (e.g., a soccer game), using training imagesfrom the same sporting event with the same teams, using training imagesfrom the same sporting event with the same teams during the same time ofday, and so on, with more detailed training generally providing moreaccurate mask results.

FIG. 4 illustrates an example input image 111 from a particular view ofan example scene, arranged in accordance with at least someimplementations of the present disclosure. In the illustrated example,scene 110 is a soccer match and input image 111 corresponds to a stadiummounted view from behind a goal of the soccer match. As discussed, inputimage 111 may be from any view of any scene such as a sporting event, anentertainment, a political event, or some other high profile event.Notably, based on attaining input images 111 from camera array 101 orscene 110, a view from a point within scene 110 may be generated as avirtual view from a virtual viewport. Such virtual views may begenerated using any suitable technique or techniques such as 3D modelconstruction (e.g., point cloud generation and texturing), viewportselection, and image reconstruction using the 3D model from theviewport. Furthermore, such virtual views offer immersive experiences ofscene 110 that are not attainable using camera array 101 alone. Usingthe techniques discussed herein, images of persons in scene 110 may besynthesized such that the persons have a pose matching a selected targetpose (e.g., a user or application selected pose) to further enhance theimmersive experience and/or for other uses as discussed herein. Asdiscussed, camera array 101 may attain any number of input images 111 ofscene 110 for use in the discussed techniques.

FIG. 5 illustrates an example background/foreground mask 112, arrangedin accordance with at least some implementations of the presentdisclosure. In the embodiment illustrated in FIG. 5, values of one areillustrated with white pixels and correspond to foreground objects orpersons 501 while values of zero are illustrated with black pixels andcorrespond to a background 502. BG/FG mask 112 may indicate any numberof foreground objects or persons of any type.

Returning to FIG. 1, input image 111 and BG/FG mask 112 are provided topose estimation module 103, which generates pose data 113 for one ormore persons represented by input image 111. Pose data 113 may begenerated using any suitable technique or techniques. In an embodiment,a CNN is applied to a foreground portion of input image 111 (as definedby BG/FG mask 112) to generate pose data 113. In an embodiment, thethree color channels of input image 111 and a fourth channel as providedby BG/FG mask 112 are provided to the CNN, which generates pose data113. In an embodiment, the CNN used to generate pose data 113 is an hourglass architecture CNN. In an embodiment, both BG/FG mask 112 and posedata 113 are generated by application of the same pretrained CNN. In anembodiment, a MaskR-CNN is applied to input image 111 to generate bothBG/FG mask 112 and pose data 113 or, optionally, just pose data 113.

Pose data 113 may indicate the pose of the person(s) in input image 111using any suitable data structure. In some embodiments, pose data 113indicate locations in input image 11 and a corresponding label for thelocation. In some embodiments, pose data 113 includes keypoint dataincluding locations of such keypoints and their corresponding body partssuch that the keypoints are connected to one another by linkstherebetween. For example, pixel locations and labels for the locationmay be provided such that each label corresponds to one of a pluralityof available labels including, for example, head, sternum, leftshoulder, right shoulder, left elbow, right elbow, left hand, righthand, left hip, right hip, pelvis, right knee, left knee, right ankle,left ankle, right foot, left foot, etc. Such labels are merelyillustrative; any number or types of labels may be used. In someembodiments, a pose skeleton (e.g., including keypoints and linkstherebetween) is provided such that the skeleton overlays the pixel dataof input image 111 and joint locations are indicated by pixel locationstherein. In some embodiments, pose data 113 may include a likelihood ofthe label corresponding to each point or pixel location.

FIG. 6 illustrates example pose data corresponding to a number ofpersons in an example input image region, arranged in accordance with atleast some implementations of the present disclosure. As shown in FIG.6, pose data 601, 602, 603, 604, 605, 606 may include pixel locationsand body feature labels corresponding to particular full body featuresof persons corresponding to pose data 601, 602, 603, 604, 605, 606. Inthe example of FIG. 6, only person 611 and several pixel locations orpoints are illustrated including right knee 612, right hip 613, andright shoulder 614 are labeled for the sake of clarity of presentation.Furthermore, bounding boxes are provided in FIG. 6 but are notnecessarily needed for the processing discussed herein.

In the example of person 611 of FIG. 6, pose data 606 and, inparticular, pose data 615 includes at least a pixel location and alocation for each of a right shoulder 614, a chest midpoint position(not labeled), a left shoulder (not labeled), a right elbow (notlabeled), a left elbow (not labeled), a right hand (not labeled), a lefthand (not labeled), a right hip 613, a left hip (not labeled), a pelvis(not labeled), a right knee 612, a left knee (not labeled), a rightankle (not labeled), a left ankle (not labeled), and one or more headfeatures (not labeled).

In the context of person 606, there is no occlusion of person 606 fromthe illustrated view such that a full body view is provided. As usedherein, the term full body view indicates a view of a pose such thatmost or all full body features are in view. In other pose positions, asillustrated further herein, one or more body parts of person 606 may beoccluded either by an external occlusion (e.g., external from person606) or a self-occlusion (e.g., one leg obscuring another leg).

Returning to FIG. 1, input image 111, pose data 113, a target pose 114,and, optionally, other data corresponding to input image 111 (asdiscussed further herein) are provided to pose synthesizer 104. Targetpose 114 may be any suitable target pose for which it is desirable aperson of input image 111 be synthesized. For example, a person may bein a running pose (e.g., a source pose) and it may be desirable totransition the person to a walking pose, a shooting pose (e.g., kickinga soccer ball), a celebratory pose, etc. In some embodiments, targetpose 114 is user selected (e.g., from a menu of available poses). Insome embodiments, target pose 114 is generated (e.g., by manipulating awire pose structure).

Using input image 111 (or a portion thereof including the person ofinterest), pose data 113, and target pose 114 (and other optional dataas needed), pose synthesizer 104 generates a synthesized pose image 115such that synthesized pose image 115 includes a representation of theperson of interest in target pose 114. In some embodiments, posesynthesizer 104 generates a body part image for the person of interestusing pose data 113 and input image 111 (or a portion thereof includingthe person of interest) such that the first body part image identifiesone of an available set of body parts for pixels representative of theperson. A body part occlusion for a body part of the representation ofthe person is then detected and rectified by identifying a second imagecorresponding to a second view of the scene having a secondrepresentation of the first body part of the person by identifyinganother image corresponding to another view of the scene having a secondrepresentation of the body part of the person and projecting the imageof the body part into the portion of the image including the person(and, optionally, projecting a body part image portion of the body partto the body part image). A geometric transformation is then applied tothe merged image including the reprojected body part to generate asynthesized image including a representation of the person in targetpose 114 such that the geometric transformation is based on the bodypart image and target pose 114.

Turning now to FIG. 2, body part isolation module 201 receives inputimage 111 or at least a portion of input image 111 corresponding to aperson of interest (e.g., a foreground portion for a person of interest)and, optionally, pose data 113, and body part isolation module 201generates a body part image 211, which includes one of an available setof body parts for pixels of the portion of input image 111. As usedherein, the term body part image or image portion indicates an image orimage portion having a body part label, from an available set of bodypart labels, for pixels thereof. For example, body part image 211 mayinclude any suitable data structure that indicates, for each pixellocation (or some pixel locations) a label of a body part from anavailable set of body parts (or a null value if no body part isdetected).

Body part isolation module 201 may generate body part image 211 usingany suitable technique or techniques. In an embodiment, body partisolation module 201 generates body part image 211 by associatingportions of an image foreground (as defined by BG/FG mask 112 withininput image 111) to the available set of body parts. That is, pixels ofthe foreground image may be associated with one of an available set ofbody parts based on pose data 113 (e.g., an overlay of the imageforeground onto pose data 113). In an embodiment, body part isolationmodule 201 applies a pretrained CNN to generate body part image 211. Inan embodiment, body part isolation module 201 and geometric transformmodule 205, discussed further below, together apply a pretrained GAN.The pretrained CNN and/or pretrained GAN may be pretrained based onpretrained image data (marked with ground truth body part data)pertinent to input images 111 at any level of granularity, as discussedabove. For example, the pretraining may be performed with trainingimages from the same sporting event being imaged, training images fromthe same sporting event with the same teams, with training images fromthe same sporting event with the same teams during the same time of day,etc. As discussed above, such detailed training may generally improvebody part labeling results.

FIG. 7 illustrates an example body part image 211 corresponding to anexample input image portion, arranged in accordance with at least someimplementations of the present disclosure. As shown in FIG. 7, for anexample image portion 741 (e.g., within image 700) corresponding to aperson (a person in a running pose in this example), body part image 211may be generated as discussed (e.g., via application of a CNN) such thatbody part image 211 has pixels labeled with body part labels such as ahead body part 701, a neck body part 702, a left upper arm body part705, left lower arm body part 704, a left hand body part 703, a torsobody part 706, a right upper leg body part 707, a right lower leg bodypart 708, a right foot body part 709, a right upper arm body part (notlabeled), a right lower arm body part (not labeled), a left hand bodypart (not labeled), a left upper leg body part (not labeled), and aright lower leg body part 711. Such body parts may correspond tokeypoints and/or links in pose data 113. In the context of FIG. 7,another body part image is illustrated with respect to a second personbut is left unlabeled for the sake of clarity of presentation.

As shown, such body part labels may be selected from an available set ofbody part labels 731 such that available set of body part labels 731includes any number (N) of available body part labels. For example, anysuitable body part labels that correspond to body parts of a person maybe used. Furthermore, as shown in FIG. 7, an external occlusion 721obscures a portion of right lower leg body part 711 and an entirety of aright foot of the person represented by image portion 741.

Returning to FIG. 2, occlusion detection module 202 receives body partimage 211 and occlusion detection module 202 detects occlusions, if any,in body part image 211. Occlusion detection module 202 may determinesuch occlusions using any suitable technique or techniques. In anembodiment, occlusion detection module 202 detects whether each labeledbody part of body part image 211 has a constant surface. In someembodiments, occlusion detection may be skipped for some body parts suchas small body parts or those that are only expected to have full or noocclusion (e.g., feet or hands). For example, a constant surfacedetection may be performed on each body part in body part image 211. Theconstant surface detection may be performed using any suitable featureof an image surface such as, for example, a texture of the surface, acolor of the surface, or directions of normal vectors from the surface.

In an embodiment, for pixels of a detected body part, particular pixelvalues are retrieved and analyzed to determine whether the body part hasa constant surface. In an embodiment, color values for pixel values areretrieved. Histograms of the color values may then be analyzed todetermine whether the body part has a constant surface color. Forexample, color histograms may be generated for a region of the body partand the region or window may be scanned across the body part. The regionbased color histograms may then be compared to determine whether thereis a color discontinuity in the body part indicative of an occlusion. Inan embodiment, color values for pixel values are retrieved and a textureof the body part is analyzed. For example, a texture value may bedetermined for a region of the body part (e.g., indicating complextexture or flat texture) and the texture values between regions may becompared to determine whether the body part has a constant surfacetexture. In an embodiment, surface normal vectors (e.g., from a pointcloud of the scene) may be analyzed to determine whether the body parthas a constant surface normal. For example, average normal vectors forregions of the body part may be generated and compared. If thedifference between any of the average normal vectors exceeds athreshold, an occlusion may be flagged. Furthermore, in someembodiments, a size and shape of the body part region may be compared toan expected size and or shape and if they are not within a threshold ofone another, an occlusion may be flagged.

FIG. 8 illustrates an example body part region 706 having an externalocclusion 810 through a portion thereof, arranged in accordance with atleast some implementations of the present disclosure. As shown in FIG.8, for body part region 706 (a torso in this example), regions 801 havea first characteristic and a region 802 has a second characteristic thatindicates region 802 is occluded (e.g., by an arm of another person).The first and second characteristics may be any suitable characteristicsdiscussed herein such as color, texture, normal vector, etc.

As shown in FIG. 8, a sub-region or window 811 may be slid across anddown body part region 706 and the characteristic for window 811 at theposition may be generated. The characteristics for each position maythen be compared to determine whether body part region 706 has anocclusion. Window 811 may be any suitable size or shape such as a 4×4pixel window, a 5×5 pixel window, or larger.

FIG. 9 illustrates an example process 900 for detecting occlusions in abody part image, arranged in accordance with at least someimplementations of the present disclosure. Process 900 may include oneor more operations 901-909 as illustrated in FIG. 9. For example,operations 901-909 may be performed by occlusion detection module 202.Process 900 begins at operation 901, where a body part represented by abody part image is selected.

Processing continues at operation 902, where the body part region (e.g.,an image region having each pixel labeled with the body part) isverified for size and shape. In an embodiment, a height and width of thebody part region are determined and compared to expected values. If anabsolute value of the difference for either the height or width exceedsa threshold, an error or occlusion is indicated at operation 907. In anembodiment, a ratio of the height and width of the body part region iscompared to an expected value. If an absolute value of the differencefor the ratio exceeds a threshold, an error or occlusion is indicated atoperation 907.

Processing continues at operation 903, where the body part region isverified for constant surface color. In an embodiment, a window isgenerated and moved across the body part image as discussed with respectto FIG. 8. For each position of the window a color value or a colorhistogram is generated. The color values or histograms for each positionmay then be compared. If a difference between the color values (e.g., adifference exceeding a threshold) or histograms (e.g., variation inhistogram peaks or other comparison) indicates a color change across thebody part region, an error or occlusion is indicated at operation 907.

Processing continues at operation 904, where the body part region isverified for constant surface texture. For example, a window may againbe moved across the body part and, for each position of the window, atexture value is generated. The texture values for each position maythen be compared. If a difference between the texture values (e.g., amax difference, a variance of the texture values, etc.) indicates atexture change (e.g., by exceeding a threshold) across the body partregion, an error or occlusion is indicated at operation 907.

Processing continues at operation 905, where the body part region isverified for constant surface normal vectors. For example, a window mayagain be moved across the body part and, for each position of thewindow, a normal vector or an average normal vector is generated. Thenormal vectors for each position may then be compared. If a differencebetween the normal vectors values (e.g., a max difference, a variance,etc.) exceeds a threshold, an error or occlusion is indicated atoperation 907.

Processing continues at decision operation 906, where a determination ismade as to whether any errors occurred at operations 902-905. If so, anindicator of an occluded body part is provided at operation 907. If not,no indicator is provided. In either case, processing continues atdecision operation 906, where a determination is made as to whether thebody part selected at operation 901 is a last body part of the body partimage. If not, processing continues at operations 901-907 as discussedabove. If so, processing continues at end operation 909.

Although discussed with respect to implementing each of a size and/orshape check, a constant surface color check, a constant surface texturecheck, and a constant surface normal vectors check, process 900 mayinclude any one or any combination of such checks.

Returning to FIG. 2, as shown, a body part indicator(s) 212 indicativeof one or more body parts having occlusions may be provided fromocclusion detection module 202. Although discussed herein with respectto size and/or shape evaluation and surface constancy detection, anysuitable techniques may be used to determine a body part having anocclusion. In an embodiment, the absence of a body part in body partimage 211 is detected and reported in body part indicator(s) 212. In anembodiment, 3D model data or other image data may be evaluated to detectand report an occluded body part.

Alternative view selector 203 receives body part indicator(s) 212 andalternative view selector 203 determines another view or views ofcorresponding to a view provided by camera array 101 that includes anunobstructed view of the body part(s) indicated by body partindicator(s) 212. Alternative view selector 203 may determine the otherview(s) using any suitable technique or techniques. For example, thediscussed segmentation, pose estimation, body part isolation, andocclusion detection processing may be performed for any number or inputimages 111 to generate corresponding BG/FG masks, pose data, and bodypart images. Input images 111, corresponding body part images, and othersuitable data structures may be stored in input images and body partimages memory store 207 for access by alternative view selector 203.Furthermore, the person of interest for the pose may be identified andlabeled in such images and so may the body parts and their status asobstructed or unobstructed, as discussed herein. Alternative viewselector 203 may then determine a closest view (e.g., a closest viewportcorresponding to a closest camera of camera array 101) that has anunobstructed view of the body part of interest for the person ofinterest. The corresponding view may be indicated by view indicator 213,which is provided to re-projection and merge module 204.

Re-projection and merge module 204 receives view indicator 213 andaccesses the pertinent input image of input images 111 or portionthereof such that at least the image of the unobstructed body part ofinterest may be re-projected to the view of the input image of interest(i.e., the input image having the obstructed body part). The image ofthe unobstructed body part may be re-projected from the second view(i.e., the view having the image of the unobstructed body part) to thefirst view (i.e., the input image having the obstructed body part) usingany suitable technique or techniques. In an embodiment, the unobstructedbody part image in the viewport of the second view is projected to thefirst view based on known locations, orientations, and calibration ofthe cameras between the views. For example, the unobstructed body partmay be scaled, rotated, and/or warped from an image plane of the secondview to an image plane of the first view based on a translation matrixor matrices (e.g., one or more camera calibration transform matrices)that may be defined between the viewports. In an embodiment, theunobstructed body part may from the second view may be translated to a3D model of the scene and then projected to the 2D viewport of thesecond view.

Although discussed herein with respect to determining a view having anunobstructed view of the body part of interest, in some embodiments, a3D model generated based on an entirety of a scene may be used togenerate the unobstructed body part image for the first view. Forexample, such 3D model construction may be used to rectify occlusionsand then a reconstructed image may be determined from the first view.

The projection of the image of the unobstructed body part may then bemerged into the portion of the input image from the first view togenerate a merged image 215 that therefore includes only unobstructedbody parts. That is, the original unobstructed body parts as provided byinput image 111 and the re-projected body part image (for the originallyobstructed body part) are merged to provide an image or image portionthat includes only unobstructed body parts.

Merged image 215, target pose 114, and pose data 113 are provided togeometric transform module 205, which generates a synthesized pose image214 having the person in target pose 114. Synthesized pose image 214 maybe generated using any suitable technique or techniques. In someembodiments, geometric transform module 205 determines a transform totranslate the image of the person from pose data 113 (e.g., a source orcurrent pose) to target pose 114. The transform may be represented inany suitable data format such as a rigid body part translation or delta.The transform may then be applied to merged image 215 to translate thebody parts thereof to target pose 114.

FIG. 10 illustrates an example geometric transformation 1001 todetermine a synthesized pose image 1003, arranged in accordance with atleast some implementations of the present disclosure. As shown in FIG.10, for a particular merged image portion 1010 representative of aperson 1005 in a source pose 1002 (e.g., illustrated as a wire structurealthough point structures as discussed herein may be used), geometrictransformation 1001 from source pose 1002 to target pose 114 may bedetermined and then applied to merged image portion 1010 to generatesynthesized pose image 1003 representative of the person 1005 in targetpose 114.

In the illustrated embodiment, source pose 1002 is representative of arunning pose while target pose 114 is representative of a celebratorypose; however, any suitable target pose 114 may be used. As discussed,geometric transformation 1001 may include a rigid body transformationdata structure that indicates location changes for each point of sourcepose 1002 to a corresponding point of target pose 114. For example,geometric transformation 1001 may indicate location changes for a headlocation, a right shoulder location, a sternum location, and so onbetween source pose 1002 and target pose 114. Geometric transformation1001 may then be applied to merged image portion 1010 to warp pixellocations in merged image portion 1010 to corresponding pixel locationsin merged image portion 1010. Furthermore, for pixel locations that donot have a one-to-one mapping, interpolation techniques may be appliedto generate pixel values (e.g., texture) at such locations. For example,pixels 1111 may be generated using interpolation techniques. It is alsonoted that some pixel values of merged image portion 1010 may bediscarded if there is not corresponding location in synthesized poseimage 1003 based on geometric transformation 1001.

Returning to FIG. 2, as shown, synthesized pose image 214 may beprovided to interpolation and background merge module 206, which mayinterpolate pixel values for synthesized pose image 214 as discussedabove. Notably, pixel values may be interpolated using bilinearinterpolation at locations between body parts (as noted at location 1011in FIG. 10). Furthermore, synthesized pose image 214 may be merged intoa background of input image 111 (based on BG/FG mask 112) such that anybackground that was previously occluded and is now needed (e.g., basedon synthesized pose image 214 having a moved body part with respect tothe person represented in input image 111) may be synthesized. Forexample, to synthesize needed background, neighboring pixels may be usedto artificially generate the synthesized background using patch copyingtechniques and/or extrapolation techniques.

As shown, the resultant synthesized pose image 115 or a portion thereofmay be transmitted for further processing and/or for presentation to auser. As used herein, the term presentation to a user includes displayvia a display device, storage to memory for immediate or later displayvia a display device, or transmission to a remote device for immediateor later display via a display device of the remote device.

FIG. 11 is a flow diagram illustrating an example process 1100 forgenerating an image of a person in a synthesized pose, arranged inaccordance with at least some implementations of the present disclosure.Process 1100 may include one or more operations 1101-1105 as illustratedin FIG. 11. Process 1100 may form at least part of an image generationprocess for generating an image of a person in an unseen pose. By way ofnon-limiting example, process 1100 may form at least part of a processas performed by apparatus 100 as discussed herein. Furthermore, process1100 will be described herein with reference to system 1200 of FIG. 12.

FIG. 12 is an illustrative diagram of an example system 1200 forgenerating an image of a person in a synthesized pose, arranged inaccordance with at least some implementations of the present disclosure.As shown in FIG. 12, system 1200 may include a central processor 1201,an image processor 1202, a memory 1203, and camera array 101. Also asshown, image processor 1202 may include or implement segmentation module102, pose estimation module 103, and pose synthesizer 104. In theexample of system 1200, memory 1203 may store image or frame data, inputimages, BG/FG masks, pose data, target pose data, synthesized poseimages, body part images, body part indicators, merged images, or anyother data discussed herein.

As shown, in some examples, one or more or portions of segmentationmodule 102, pose estimation module 103, and pose synthesizer 104 areimplemented via image processor 1202. In other examples, one or more orportions of segmentation module 102, pose estimation module 103, andpose synthesizer 104 are implemented via central processor 1201, animage processing unit, an image processing pipeline, an image signalprocessor, or the like. In some examples, one or more or portions ofsegmentation module 102, pose estimation module 103, and posesynthesizer 104 are implemented in hardware as a system-on-a-chip (SoC).In some examples, one or more or portions of segmentation module 102,pose estimation module 103, and pose synthesizer 104 are implemented inhardware via a FPGA.

Image processor 1202 may include any number and type of image orgraphics processing units that may provide the operations as discussedherein. Such operations may be implemented via software or hardware or acombination thereof. For example, image processor 1202 may includecircuitry dedicated to manipulate and/or analyze images obtained frommemory 1203. Central processor 1201 may include any number and type ofprocessing units or modules that may provide control and other highlevel functions for system 1200 and/or provide any operations asdiscussed herein. Memory 1203 may be any type of memory such as volatilememory (e.g., Static Random Access Memory (SRAM), Dynamic Random AccessMemory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.),and so forth. In a non-limiting example, memory 1203 may be implementedby cache memory. In an embodiment, one or more or portions ofsegmentation module 102, pose estimation module 103, and posesynthesizer 104 are implemented via an execution unit (EU) of imageprocessor 1202. The EU may include, for example, programmable logic orcircuitry such as a logic core or cores that may provide a wide array ofprogrammable logic functions. In an embodiment, one or more or portionsof segmentation module 102, pose estimation module 103, and posesynthesizer 104 are implemented via dedicated hardware such as fixedfunction circuitry or the like. Fixed function circuitry may includededicated logic or circuitry and may provide a set of fixed functionentry points that may map to the dedicated logic for a fixed purpose orfunction.

Returning to discussion of FIG. 11, process 1100 begins at operation1101, where pose data corresponding to a person represented in a firstimage are received such that the first image corresponds to a first viewof a scene including the person. In some embodiments, process 1100further includes performing background and foreground segmentation on aninput image corresponding to the first image by applying a convolutionalneural network to the input image to generate a background andforeground mask that includes a first value for background pixels of theinput image and a second value for foreground pixels of the input image.Furthermore, process 1100 may include applying a second convolutionalneural network to at least a portion of the input image corresponding toan image foreground based on the background and foreground mask togenerate the pose data. For example, the pose data may include at leastlocations of an available set of body parts for the person.

Processing continues at operation 1102, where a first body part image isgenerated for the person using at least the first image such that thefirst body part image identifies one of an available set of body partsfor pixels of the first image. The available set of body parts mayinclude any body parts corresponding to a human. The first body partimage may be generated using any suitable technique or techniques suchas application of a convolutional neural network.

Processing continues at operation 1103, where a body part occlusion isdetected for a first body part of the representation of the person fromthe first view. The body part occlusion may be detected using anysuitable technique or techniques. In an embodiment, detecting the bodypart occlusion for the first body part includes performing constantsurface detection on a first body part region corresponding to the firstbody part in the first body part image to detect the first body partregion comprises a non-constant image surface. In an embodiment, theconstant surface detection includes texture detection and detection ofthe non-constant image surface includes detection of a texture changebetween sub-regions of the first body part region exceeds a threshold.In an embodiment, the constant surface detection includes colordetection and detection of the non-constant image surface includesdetection of a color histogram change between sub-regions of the firstbody part region. In an embodiment, the constant surface detectionincludes normal vector change detection and detection of thenon-constant image surface includes detection of an average normalvector change between sub-regions of the first body part region. In anembodiment, the constant surface detection includes edge detection anddetection of the non-constant image surface includes detection of one ormore edges in the first body part region.

Processing continues at operation 1104, where a second imagecorresponding to a second view of the scene having a secondrepresentation of the first body part of the person is identified. Thesecond image may be identified using any suitable technique ortechniques. For example, a body part image may be formed for the personin any number of images (corresponding to any number of different views)and the pertinent body part may be analyzed for occlusions. A closestview to the current view having a non-occluded representation of thebody part may then be determined (e.g., based on camera distance).

Processing continues at operation 1105, where the second representationof the first body part in the second image is projected to the firstimage to generate a merged image. The second representation of the firstbody part in the second image may be projected to the first image usingany suitable technique or techniques such as image transform operationsbased on transform matrices that are in turn based on the locations andcalibration parameters of the cameras used to capture the input imagesfor the first and second views.

Processing continues at operation 1106, where a geometric transformationis applied to the merged image based at least on the pose data and atarget pose to generate a synthesized image comprising a representationof the person in the target pose. The geometric transformation may beapplied using any suitable technique or techniques. In an embodiment,the geometric transformation is determined as a rigid body transformthat transforms the pose data to the target pose and the geometrictransformation is then applied to the merged image to warp the mergedimage into the synthesized image. In some embodiments, generating thefirst body part image and applying the geometric transformation areperformed at least in part by a pretrained generative adversarialnetwork. In some embodiments, generating the first body part imageincludes associating portions of the image foreground to the availableset of body parts.

In some embodiments, process 1100 further includes performinginterpolation between body part segments of the synthesized image tomerge the body part segments and merging a resultant final pose into abackground to generate a final synthesized image, and presenting thefinal synthesized image to a user. Such a final synthesized image havinga representation of a person in an unseen human pose may be used in anysuitable application as discussed herein.

Process 1100 may be repeated any number of times either in series or inparallel for any number of input images, persons within an input image,target poses, or the like. Process 1100 provides for synthesized poseestimation that is automated, computationally efficient, and accurate inreal world, multiple person, and low resolution environments.

Process 1100 may be implemented by any suitable device, system, orplatform such as those discussed herein. In an embodiment, process 1100is implemented by an apparatus having a memory to store images, as wellas any other discussed data structure, and a processor to performoperations 1101-1105. In an embodiment, the memory and the processor areimplemented via a monolithic field programmable gate array integratedcircuit. As used herein, the term monolithic indicates a device that isdiscrete from other devices, although it may be coupled to other devicesfor communication and power supply.

Various components of the systems described herein may be implemented insoftware, firmware, and/or hardware and/or any combination thereof. Forexample, various components of the devices or systems discussed hereinmay be provided, at least in part, by hardware of a computingSystem-on-a-Chip (SoC) such as may be found in a computing system suchas, for example, a smart phone. Those skilled in the art may recognizethat systems described herein may include additional components thathave not been depicted in the corresponding figures. For example, thesystems discussed herein may include additional components that have notbeen depicted in the interest of clarity.

While implementation of the example processes discussed herein mayinclude the undertaking of all operations shown in the orderillustrated, the present disclosure is not limited in this regard and,in various examples, implementation of the example processes herein mayinclude only a subset of the operations shown, operations performed in adifferent order than illustrated, or additional operations.

In addition, any one or more of the operations discussed herein may beundertaken in response to instructions provided by one or more computerprogram products. Such program products may include signal bearing mediaproviding instructions that, when executed by, for example, a processor,may provide the functionality described herein. The computer programproducts may be provided in any form of one or more machine-readablemedia. Thus, for example, a processor including one or more graphicsprocessing unit(s) or processor core(s) may undertake one or more of theblocks of the example processes herein in response to program codeand/or instructions or instruction sets conveyed to the processor by oneor more machine-readable media. In general, a machine-readable mediummay convey software in the form of program code and/or instructions orinstruction sets that may cause any of the devices and/or systemsdescribed herein to implement at least portions of the devices orsystems, or any other module or component as discussed herein.

As used in any implementation described herein, the term “module” refersto any combination of software logic, firmware logic, hardware logic,and/or circuitry configured to provide the functionality describedherein. The software may be embodied as a software package, code and/orinstruction set or instructions, and “hardware”, as used in anyimplementation described herein, may include, for example, singly or inany combination, hardwired circuitry, programmable circuitry, statemachine circuitry, fixed function circuitry, execution unit circuitry,and/or firmware that stores instructions executed by programmablecircuitry. The modules may, collectively or individually, be embodied ascircuitry that forms part of a larger system, for example, an integratedcircuit (IC), system on-chip (SoC), and so forth.

FIG. 13 is an illustrative diagram of an example system 1300, arrangedin accordance with at least some implementations of the presentdisclosure. In various implementations, system 1300 may be a mobiledevice system although system 1300 is not limited to this context. Forexample, system 1300 may be incorporated into a personal computer (PC),laptop computer, ultra-laptop computer, tablet, touch pad, portablecomputer, handheld computer, palmtop computer, personal digitalassistant (PDA), cellular telephone, combination cellular telephone/PDA,television, smart device (e.g., smart phone, smart tablet or smarttelevision), mobile internet device (MID), messaging device, datacommunication device, cameras (e.g. point-and-shoot cameras, super-zoomcameras, digital single-lens reflex (DSLR) cameras), a surveillancecamera, a surveillance system including a camera, and so forth.

In various implementations, system 1300 includes a platform 1302 coupledto a display 1320. Platform 1302 may receive content from a contentdevice such as content services device(s) 1330 or content deliverydevice(s) 1340 or other content sources such as image sensors 1319. Forexample, platform 1302 may receive image data as discussed herein fromimage sensors 1319 or any other content source. A navigation controller1350 including one or more navigation features may be used to interactwith, for example, platform 1302 and/or display 1320. Each of thesecomponents is described in greater detail below.

In various implementations, platform 1302 may include any combination ofa chipset 1305, processor 1310, memory 1312, antenna 1313, storage 1314,graphics subsystem 1315, applications 1316, image signal processor 1317and/or radio 1318. Chipset 1305 may provide intercommunication amongprocessor 1310, memory 1312, storage 1314, graphics subsystem 1315,applications 1316, image signal processor 1317 and/or radio 1318. Forexample, chipset 1305 may include a storage adapter (not depicted)capable of providing intercommunication with storage 1314.

Processor 1310 may be implemented as a Complex Instruction Set Computer(CISC) or Reduced Instruction Set Computer (RISC) processors, x86instruction set compatible processors, multi-core, or any othermicroprocessor or central processing unit (CPU). In variousimplementations, processor 1310 may be dual-core processor(s), dual-coremobile processor(s), and so forth.

Memory 1312 may be implemented as a volatile memory device such as, butnot limited to, a Random Access Memory (RAM), Dynamic Random AccessMemory (DRAM), or Static RAM (SRAM).

Storage 1314 may be implemented as a non-volatile storage device suchas, but not limited to, a magnetic disk drive, optical disk drive, tapedrive, an internal storage device, an attached storage device, flashmemory, battery backed-up SDRAM (synchronous DRAM), and/or a networkaccessible storage device. In various implementations, storage 1314 mayinclude technology to increase the storage performance enhancedprotection for valuable digital media when multiple hard drives areincluded, for example.

Image signal processor 1317 may be implemented as a specialized digitalsignal processor or the like used for image processing. In someexamples, image signal processor 1317 may be implemented based on asingle instruction multiple data or multiple instruction multiple dataarchitecture or the like. In some examples, image signal processor 1317may be characterized as a media processor. As discussed herein, imagesignal processor 1317 may be implemented based on a system on a chiparchitecture and/or based on a multi-core architecture.

Graphics subsystem 1315 may perform processing of images such as stillor video for display. Graphics subsystem 1315 may be a graphicsprocessing unit (GPU) or a visual processing unit (VPU), for example. Ananalog or digital interface may be used to communicatively couplegraphics subsystem 1315 and display 1320. For example, the interface maybe any of a High-Definition Multimedia Interface, DisplayPort, wirelessHDMI, and/or wireless HD compliant techniques. Graphics subsystem 1315may be integrated into processor 1310 or chipset 1305. In someimplementations, graphics subsystem 1315 may be a stand-alone devicecommunicatively coupled to chipset 1305.

The graphics and/or video processing techniques described herein may beimplemented in various hardware architectures. For example, graphicsand/or video functionality may be integrated within a chipset.Alternatively, a discrete graphics and/or video processor may be used.As still another implementation, the graphics and/or video functions maybe provided by a general purpose processor, including a multi-coreprocessor. In further embodiments, the functions may be implemented in aconsumer electronics device.

Radio 1318 may include one or more radios capable of transmitting andreceiving signals using various suitable wireless communicationstechniques. Such techniques may involve communications across one ormore wireless networks. Example wireless networks include (but are notlimited to) wireless local area networks (WLANs), wireless personal areanetworks (WPANs), wireless metropolitan area network (WMANs), cellularnetworks, and satellite networks. In communicating across such networks,radio 1318 may operate in accordance with one or more applicablestandards in any version.

In various implementations, display 1320 may include any television typemonitor or display. Display 1320 may include, for example, a computerdisplay screen, touch screen display, video monitor, television-likedevice, and/or a television. Display 1320 may be digital and/or analog.In various implementations, display 1320 may be a holographic display.Also, display 1320 may be a transparent surface that may receive avisual projection. Such projections may convey various forms ofinformation, images, and/or objects. For example, such projections maybe a visual overlay for a mobile augmented reality (MAR) application.Under the control of one or more software applications 1316, platform1302 may display user interface 1322 on display 1320.

In various implementations, content services device(s) 1330 may behosted by any national, international and/or independent service andthus accessible to platform 1302 via the Internet, for example. Contentservices device(s) 1330 may be coupled to platform 1302 and/or todisplay 1320. Platform 1302 and/or content services device(s) 1330 maybe coupled to a network 1360 to communicate (e.g., send and/or receive)media information to and from network 1360. Content delivery device(s)1340 also may be coupled to platform 1302 and/or to display 1320.

Image sensors 1319 may include any suitable image sensors that mayprovide image data based on a scene. For example, image sensors 1319 mayinclude a semiconductor charge coupled device (CCD) based sensor, acomplimentary metal-oxide-semiconductor (CMOS) based sensor, an N-typemetal-oxide-semiconductor (NMOS) based sensor, or the like. For example,image sensors 1319 may include any device that may detect information ofa scene to generate image data.

In various implementations, content services device(s) 1330 may includea cable television box, personal computer, network, telephone, Internetenabled devices or appliance capable of delivering digital informationand/or content, and any other similar device capable ofuni-directionally or bi-directionally communicating content betweencontent providers and platform 1302 and/display 1320, via network 1360or directly. It will be appreciated that the content may be communicateduni-directionally and/or bi-directionally to and from any one of thecomponents in system 1300 and a content provider via network 1360.Examples of content may include any media information including, forexample, video, music, medical and gaming information, and so forth.

Content services device(s) 1330 may receive content such as cabletelevision programming including media information, digital information,and/or other content. Examples of content providers may include anycable or satellite television or radio or Internet content providers.The provided examples are not meant to limit implementations inaccordance with the present disclosure in any way.

In various implementations, platform 1302 may receive control signalsfrom navigation controller 1350 having one or more navigation features.The navigation features of navigation controller 1350 may be used tointeract with user interface 1322, for example. In various embodiments,navigation controller 1350 may be a pointing device that may be acomputer hardware component (specifically, a human interface device)that allows a user to input spatial (e.g., continuous andmulti-dimensional) data into a computer. Many systems such as graphicaluser interfaces (GUI), and televisions and monitors allow the user tocontrol and provide data to the computer or television using physicalgestures.

Movements of the navigation features of navigation controller 1350 maybe replicated on a display (e.g., display 1320) by movements of apointer, cursor, focus ring, or other visual indicators displayed on thedisplay. For example, under the control of software applications 1316,the navigation features located on navigation controller 1350 may bemapped to virtual navigation features displayed on user interface 1322,for example. In various embodiments, navigation controller 1350 may notbe a separate component but may be integrated into platform 1302 and/ordisplay 1320. The present disclosure, however, is not limited to theelements or in the context shown or described herein.

In various implementations, drivers (not shown) may include technologyto enable users to instantly turn on and off platform 1302 like atelevision with the touch of a button after initial boot-up, whenenabled, for example. Program logic may allow platform 1302 to streamcontent to media adaptors or other content services device(s) 1330 orcontent delivery device(s) 1340 even when the platform is turned “off.”In addition, chipset 1305 may include hardware and/or software supportfor 5.1 surround sound audio and/or high definition 7.1 surround soundaudio, for example. Drivers may include a graphics driver for integratedgraphics platforms. In various embodiments, the graphics driver maycomprise a peripheral component interconnect (PCI) Express graphicscard.

In various implementations, any one or more of the components shown insystem 1300 may be integrated. For example, platform 1302 and contentservices device(s) 1330 may be integrated, or platform 1302 and contentdelivery device(s) 1340 may be integrated, or platform 1302, contentservices device(s) 1330, and content delivery device(s) 1340 may beintegrated, for example. In various embodiments, platform 1302 anddisplay 1320 may be an integrated unit. Display 1320 and content servicedevice(s) 1330 may be integrated, or display 1320 and content deliverydevice(s) 1340 may be integrated, for example. These examples are notmeant to limit the present disclosure.

In various embodiments, system 1300 may be implemented as a wirelesssystem, a wired system, or a combination of both. When implemented as awireless system, system 1300 may include components and interfacessuitable for communicating over a wireless shared media, such as one ormore antennas, transmitters, receivers, transceivers, amplifiers,filters, control logic, and so forth. An example of wireless sharedmedia may include portions of a wireless spectrum, such as the RFspectrum and so forth. When implemented as a wired system, system 1300may include components and interfaces suitable for communicating overwired communications media, such as input/output (I/O) adapters,physical connectors to connect the I/O adapter with a correspondingwired communications medium, a network interface card (NIC), disccontroller, video controller, audio controller, and the like. Examplesof wired communications media may include a wire, cable, metal leads,printed circuit board (PCB), backplane, switch fabric, semiconductormaterial, twisted-pair wire, co-axial cable, fiber optics, and so forth.

Platform 1302 may establish one or more logical or physical channels tocommunicate information. The information may include media informationand control information. Media information may refer to any datarepresenting content meant for a user. Examples of content may include,for example, data from a voice conversation, videoconference, streamingvideo, electronic mail (“email”) message, voice mail message,alphanumeric symbols, graphics, image, video, text and so forth. Datafrom a voice conversation may be, for example, speech information,silence periods, background noise, comfort noise, tones and so forth.Control information may refer to any data representing commands,instructions or control words meant for an automated system. Forexample, control information may be used to route media informationthrough a system, or instruct a node to process the media information ina predetermined manner. The embodiments, however, are not limited to theelements or in the context shown or described in FIG. 13.

As described above, system 1300 may be embodied in varying physicalstyles or form factors. FIG. 14 illustrates an example small form factordevice 1400, arranged in accordance with at least some implementationsof the present disclosure. In some examples, system 1300 may beimplemented via device 1400. In other examples, other systems,components, or modules discussed herein or portions thereof may beimplemented via device 1400. In various embodiments, for example, device1400 may be implemented as a mobile computing device a having wirelesscapabilities. A mobile computing device may refer to any device having aprocessing system and a mobile power source or supply, such as one ormore batteries, for example.

Examples of a mobile computing device may include a personal computer(PC), laptop computer, ultra-laptop computer, tablet, touch pad,portable computer, handheld computer, palmtop computer, personal digitalassistant (PDA), cellular telephone, combination cellular telephone/PDA,smart device (e.g., smartphone, smart tablet or smart mobiletelevision), mobile internet device (MID), messaging device, datacommunication device, cameras (e.g. point-and-shoot cameras, super-zoomcameras, digital single-lens reflex (DSLR) cameras), and so forth.

Examples of a mobile computing device also may include computers thatare arranged to be implemented by a motor vehicle or robot, or worn by aperson, such as wrist computers, finger computers, ring computers,eyeglass computers, belt-clip computers, arm-band computers, shoecomputers, clothing computers, and other wearable computers. In variousembodiments, for example, a mobile computing device may be implementedas a smartphone capable of executing computer applications, as well asvoice communications and/or data communications. Although someembodiments may be described with a mobile computing device implementedas a smartphone by way of example, it may be appreciated that otherembodiments may be implemented using other wireless mobile computingdevices as well. The embodiments are not limited in this context.

As shown in FIG. 14, device 1400 may include a housing with a front 1401and a back 1402. Device 1400 includes a display 1404, an input/output(I/O) device 1406, a color camera 1421, a color camera 1422, an infraredtransmitter 1423, and an integrated antenna 1408. In some embodiments,color camera 1421 and color camera 1422 attain planar images asdiscussed herein. In some embodiments, device 1400 does not includecolor camera 1421 and 1422 and device 1400 attains input image data(e.g., any input image data discussed herein) from another device.Device 1400 also may include navigation features 1412. I/O device 1406may include any suitable I/O device for entering information into amobile computing device. Examples for I/O device 1406 may include analphanumeric keyboard, a numeric keypad, a touch pad, input keys,buttons, switches, microphones, speakers, voice recognition device andsoftware, and so forth. Information also may be entered into device 1400by way of microphone (not shown), or may be digitized by a voicerecognition device. As shown, device 1400 may include color cameras1421, 1422, and a flash 1410 integrated into back 1402 (or elsewhere) ofdevice 1400. In other examples, color cameras 1421, 1422, and flash 1410may be integrated into front 1401 of device 1400 or both front and backsets of cameras may be provided. Color cameras 1421, 1422 and a flash1410 may be components of a camera module to originate color image datawith IR texture correction that may be processed into an image orstreaming video that is output to display 1404 and/or communicatedremotely from device 1400 via antenna 1408 for example.

Various embodiments may be implemented using hardware elements, softwareelements, or a combination of both. Examples of hardware elements mayinclude processors, microprocessors, circuits, circuit elements (e.g.,transistors, resistors, capacitors, inductors, and so forth), integratedcircuits, application specific integrated circuits (ASIC), programmablelogic devices (PLD), digital signal processors (DSP), field programmablegate array (FPGA), logic gates, registers, semiconductor device, chips,microchips, chip sets, and so forth. Examples of software may includesoftware components, programs, applications, computer programs,application programs, system programs, machine programs, operatingsystem software, middleware, firmware, software modules, routines,subroutines, functions, methods, procedures, software interfaces,application program interfaces (API), instruction sets, computing code,computer code, code segments, computer code segments, words, values,symbols, or any combination thereof. Determining whether an embodimentis implemented using hardware elements and/or software elements may varyin accordance with any number of factors, such as desired computationalrate, power levels, heat tolerances, processing cycle budget, input datarates, output data rates, memory resources, data bus speeds and otherdesign or performance constraints.

One or more aspects of at least one embodiment may be implemented byrepresentative instructions stored on a machine-readable medium whichrepresents various logic within the processor, which when read by amachine causes the machine to fabricate logic to perform the techniquesdescribed herein. Such representations, known as IP cores may be storedon a tangible, machine readable medium and supplied to various customersor manufacturing facilities to load into the fabrication machines thatactually make the logic or processor.

While certain features set forth herein have been described withreference to various implementations, this description is not intendedto be construed in a limiting sense. Hence, various modifications of theimplementations described herein, as well as other implementations,which are apparent to persons skilled in the art to which the presentdisclosure pertains are deemed to lie within the spirit and scope of thepresent disclosure.

In one or more first embodiments, a method for generating an image of aperson in a synthesized pose comprises receiving pose data correspondingto a person represented in a first image, wherein the first imagecorresponds to a first view of a scene comprising the person, generatinga first body part image for the person using at least the first image,wherein the first body part image identifies one of an available set ofbody parts for pixels of the first image, detecting a body partocclusion for a first body part of the representation of the person fromthe first view, identifying a second image corresponding to a secondview of the scene having a second representation of the first body partof the person, projecting the second representation of the first bodypart in the second image to the first image to generate a merged image,and applying a geometric transformation to the merged image based atleast on the pose data and a target pose to generate a synthesized imagecomprising a representation of the person in the target pose.

In one or more second embodiments, further to the first embodiments,detecting the body part occlusion for the first body part comprisesperforming constant surface detection on a first body part regioncorresponding to the first body part in the first body part image todetect the first body part region comprises a non-constant imagesurface.

In one or more third embodiments, further to the first or secondembodiments, the constant surface detection comprises texture detectionand detection of the non-constant image surface comprises detection of atexture change between sub-regions of the first body part region exceedsa threshold.

In one or more fourth embodiments, further to the first through thirdembodiments, the constant surface detection comprises color detectionand detection of the non-constant image surface comprises detection of acolor histogram change between sub-regions of the first body partregion.

In one or more fifth embodiments, further to the first through fourthembodiments, the constant surface detection comprises normal vectorchange detection and detection of the non-constant image surfacecomprises detection of an average normal vector change betweensub-regions of the first body part region.

In one or more sixth embodiments, further to the first through fifthembodiments, the constant surface detection comprises edge detection anddetection of the non-constant image surface comprises detection of oneor more edges in the first body part region.

In one or more seventh embodiments, further to the first through sixthembodiments, the method further comprises performing background andforeground segmentation on an input image corresponding to the firstimage by applying a convolutional neural network to the input image togenerate a background and foreground mask comprising a first value forbackground pixels of the input image and a second value for foregroundpixels of the input image.

In one or more eighth embodiments, further to the first through seventhembodiments, the method further comprises applying a secondconvolutional neural network to at least a portion of the input imagecorresponding to an image foreground based on the background andforeground mask to generate the pose data, wherein the pose datacomprises at least locations of the available set of body parts for theperson.

In one or more ninth embodiments, further to the first through eighthembodiments, generating the first body part image comprises associatingportions of the image foreground to the available set of body parts.

In one or more tenth embodiments, further to the first through ninthembodiments, the method further comprises performing interpolationbetween body part segments of the synthesized image to merge the bodypart segments and merging a resultant final pose into a background togenerate a final synthesized image and presenting the final synthesizedimage to a user.

In one or more eleventh embodiments, a device or system includes amemory and a processor to perform a method according to any one of theabove embodiments.

In one or more twelfth embodiments, at least one machine readable mediumincludes a plurality of instructions that in response to being executedon a computing device, cause the computing device to perform a methodaccording to any one of the above embodiments.

In one or more thirteenth embodiments, an apparatus includes means forperforming a method according to any one of the above embodiments.

It will be recognized that the embodiments are not limited to theembodiments so described, but can be practiced with modification andalteration without departing from the scope of the appended claims. Forexample, the above embodiments may include specific combination offeatures. However, the above embodiments are not limited in this regardand, in various implementations, the above embodiments may include theundertaking only a subset of such features, undertaking a differentorder of such features, undertaking a different combination of suchfeatures, and/or undertaking additional features than those featuresexplicitly listed. The scope of the embodiments should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. An apparatus for generating an image of a personin a synthesized pose comprising: a memory to store pose datacorresponding to a person represented in a first image, wherein thefirst image corresponds to a first view of a scene comprising theperson; and a processor coupled to the memory, the processor to:generate a first body part image for the person using at least the firstimage, wherein the first body part image identifies one of an availableset of body parts for pixels of the first image; detect a body partocclusion for a first body part of the representation of the person fromthe first view; identify a second image corresponding to a second viewof the scene having a second representation of the first body part ofthe person; project the second representation of the first body part inthe second image to the first image to generate a merged image; andapply a geometric transformation to the merged image based at least onthe pose data and a target pose to generate a synthesized imagecomprising a representation of the person in the target pose.
 2. Theapparatus of claim 1, wherein the processor to detect the body partocclusion for the first body part comprises the processor to performconstant surface detection on a first body part region corresponding tothe first body part in the first body part image to detect the firstbody part region comprises a non-constant image surface.
 3. Theapparatus of claim 2, wherein the constant surface detection comprisestexture detection and detection of the non-constant image surfacecomprises detection of a texture change between sub-regions of the firstbody part region exceeds a threshold.
 4. The apparatus of claim 2,wherein the constant surface detection comprises color detection anddetection of the non-constant image surface comprises detection of acolor histogram change between sub-regions of the first body partregion.
 5. The apparatus of claim 2, wherein the constant surfacedetection comprises normal vector change detection and detection of thenon-constant image surface comprises detection of an average normalvector change between sub-regions of the first body part region.
 6. Theapparatus of claim 2, wherein the constant surface detection comprisesedge detection and detection of the non-constant image surface comprisesdetection of one or more edges in the first body part region.
 7. Theapparatus of claim 1, the processor further to: perform background andforeground segmentation on an input image corresponding to the firstimage via application of a convolutional neural network to the inputimage to generate a background and foreground mask comprising a firstvalue for background pixels of the input image and a second value forforeground pixels of the input image.
 8. The apparatus of claim 7, theprocessor further to: apply a second convolutional neural network to atleast a portion of the input image corresponding to an image foregroundbased on the background and foreground mask to generate the pose data,wherein the pose data comprises at least locations of the available setof body parts for the person.
 9. The apparatus of claim 7, wherein theprocessor to generate the first body part image comprises the processorto associate portions of the image foreground to the available set ofbody parts.
 10. The apparatus of claim 1, the processor further to:perform interpolation between body part segments of the synthesizedimage to merge the body part segments and merge a resultant final poseinto a background to generate a final synthesized image; and present thefinal synthesized image to a user.
 11. A method for generating an imageof a person in a synthesized pose comprising: receiving pose datacorresponding to a person represented in a first image, wherein thefirst image corresponds to a first view of a scene comprising theperson; generating a first body part image for the person using at leastthe first image, wherein the first body part image identifies one of anavailable set of body parts for pixels of the first image; detecting abody part occlusion for a first body part of the representation of theperson from the first view; identifying a second image corresponding toa second view of the scene having a second representation of the firstbody part of the person; projecting the second representation of thefirst body part in the second image to the first image to generate amerged image; and applying a geometric transformation to the mergedimage based at least on the pose data and a target pose to generate asynthesized image comprising a representation of the person in thetarget pose.
 12. The method of claim 11, wherein detecting the body partocclusion for the first body part comprises performing constant surfacedetection on a first body part region corresponding to the first bodypart in the first body part image to detect the first body part regioncomprises a non-constant image surface.
 13. The method of claim 12,wherein the constant surface detection comprises texture detection anddetection of the non-constant image surface comprises detection of atexture change between sub-regions of the first body part region exceedsa threshold.
 14. The method of claim 12, wherein the constant surfacedetection comprises color detection and detection of the non-constantimage surface comprises detection of a color histogram change betweensub-regions of the first body part region.
 15. The method of claim 11,further comprising: performing background and foreground segmentation onan input image corresponding to the first image by applying aconvolutional neural network to the input image to generate a backgroundand foreground mask comprising a first value for background pixels ofthe input image and a second value for foreground pixels of the inputimage; and applying a second convolutional neural network to at least aportion of the input image corresponding to an image foreground based onthe background and foreground mask to generate the pose data, whereinthe pose data comprises at least locations of the available set of bodyparts for the person.
 16. The method of claim 15, wherein saidgenerating the first body part image comprises associating portions ofthe image foreground to the available set of body parts.
 17. At leastone non-transitory machine readable medium comprising a plurality ofinstructions that, in response to being executed on a computing device,cause the computing device to generate an image of a person in asynthesized pose by: receiving pose data corresponding to a personrepresented in a first image, wherein the first image corresponds to afirst view of a scene comprising the person; generating a first bodypart image for the person using at least the first image, wherein thefirst body part image identifies one of an available set of body partsfor pixels of the first image; detecting a body part occlusion for afirst body part of the representation of the person from the first view;identifying a second image corresponding to a second view of the scenehaving a second representation of the first body part of the person;projecting the second representation of the first body part in thesecond image to the first image to generate a merged image; and applyinga geometric transformation to the merged image based at least on thepose data and a target pose to generate a synthesized image comprising arepresentation of the person in the target pose.
 18. The non-transitorymachine readable medium of claim 17, wherein detecting the body partocclusion for the first body part comprises performing constant surfacedetection on a first body part region corresponding to the first bodypart in the first body part image to detect the first body part regioncomprises a non-constant image surface.
 19. The non-transitory machinereadable medium of claim 18, wherein the constant surface detectioncomprises texture detection and detection of the non-constant imagesurface comprises detection of a texture change between sub-regions ofthe first body part region exceeds a threshold.
 20. The non-transitorymachine readable medium of claim 18, wherein the constant surfacedetection comprises color detection and detection of the non-constantimage surface comprises detection of a color histogram change betweensub-regions of the first body part region.
 21. The non-transitorymachine readable medium of claim 17, further comprising instructionsthat, in response to being executed on the computing device, cause thecomputing device to generate the image of the person in the synthesizedpose by: performing background and foreground segmentation on an inputimage corresponding to the first image by applying a convolutionalneural network to the input image to generate a background andforeground mask comprising a first value for background pixels of theinput image and a second value for foreground pixels of the input image;and applying a second convolutional neural network to at least a portionof the input image corresponding to an image foreground based on thebackground and foreground mask to generate the pose data, wherein thepose data comprises at least locations of the available set of bodyparts for the person.
 22. The non-transitory machine readable medium ofclaim 21, wherein said generating the first body part image comprisesassociating portions of the image foreground to the available set ofbody parts.